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PREFACE 


IDA  prepared  this  paper  in  partial  fulfillment  of  task  order  entitled  ‘Target 
Acquisition  and  Search  Studies,”  for  Mr.  Walter  Hollis,  Deputy  Under  Secretary  of  the 
Army  [DUS A  (OR)],  and  Mr.  Raymond  S.  Balcerak,  Program  Manager,  ARPA  Infrared 
Focal  Plane  Array  Producibility  Microelectronics  Technology  Office. 

IDA  performed  this  work  under  the  auspices  of  Mr.  Hollis’  ACQSIM  (Acquisition 
Simulation  Working  Group)  activity.  In  an  earlier  ACQSIM  forum,  Mr.  David  Dixon  of 
TRAC-WSMR  (TRADOC  Missile  Command-White  Sands  Missile  Range)  presented  an 
analysis  of  alternative  formulations  of  observer  variability  in  target  acquisition  simulations. 
That  analysis  demonstrated  the  importance  of  resolving  these  long-standing  issues,  and 
motivated  the  present  work. 

The  high  quality  target  acquisition  performance  data  which  the  Visionics  Division 
of  NVESD  obtained  in  their  Target  Acquisition  Model  Improvement  Program  (TAMIP) 
Phase  1  target  acquisition  tests  were  indispensable  to  the  completion  of  this  study. 
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EXECUTIVE  SUMMARY 


The  Night  Vision  Standard  ModeP  defines  Poo  as  the  fraction  of  a  large  sample  of 
observers,  drawn  from  the  standard  ensemble,  that  will,  given  unlimited  time,  eventually 
detect  a  given  target. 

Even  given  that  this  definition  is  understood,  what  is  the  meaning  of  Pool  The 
question  has  been  asked  frequently  since  the  introduction  of  the  Standard  Model.  At  first 
glance,  the  question  seems  at  best  irrelevant;  at  worst,  without  content.  After  all,  given  the 
value  of  Poo,  we  can  by  definition  predict  (up  to  statistical  uncertainties)  how  many 
observers  out  of  a  given  sample  wiU  eventually  succeed  in  a  specific  target  acquisition  task. 

Nevertheless,  different  interpretations  are  possible.  A  seminal  work^  on  the  subject 
discussed  several  of  them  at  length  and  proposed  possible  implications.  Here  are  two  of 
these  interpretations: 

•  All  observers  are  equivalent  in  a  statistical  sense,  but  their  responses  to  a  given 
target  are  stochastic.  Thus,  if  Poo  is  50  percent,  then  any  given  observer  has  a 
50-percent  chance  of  detecting  it.  This  is  independent  of  that  observer’s 
chance  of  detecting  any  other  Poo  =  50  percent  target. 

•  All  observers  are  strictly  ordered  in  target  acquisition  competence.  There  is  no 
stochastic  element  at  all;  the  detection  “probability”  is  really  just  the  fraction  of 
the  observer  ensemble  that  is  sufficiently  competent  to  accomplish  the  detection 
of  the  given  target.  In  this  case,  every  Poo  =  50  percent  target  is  always 
detected  by  above-average  observers  and  never  detected  by  below-average 
ones. 

It  is  well  known  that  neither  of  these  extreme  interpretations  is  literally  true.  While 
some  observers  are  certainly  better  than  others,  strict  adherence  to  hierarchical  ordering  is 
not  observed.  So  some  sort  of  hybrid  model  seems  to  be  called  for.  But  to  date  no  such 
approach  has  been  validated,  nor  is  there  even  an  understanding  of  which  approach  is 
closer  to  reality. 


^  Ratches  et  al. 

2  Marta  Kowalczyk  and  Stanley  Rotman,  Extending  the  CNVEO  Search  Model  to  the  Multitarget 
Environment,  Institute  for  Defense  Analyses,  IDA  Paper  P-2022, 1987. 
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It  is  easy  to  see  why  this  matter  has  remained  unresolved.  For  a  one-on-one 
engagement  the  question  of  interpretation  does  not  arise — both  approaches  give  the  same 
answer.  Thus,  there  has  in  the  past  been  only  theoretical  interest  in  determining  the  correct 
form  of  such  a  hybrid  model.  In  fact,  there  has  been  little  concern  that  the  two  most 
commonly  used  Army  combat  models,  CASTFOREM  and  JANUS,  adopt  opposed  views. 
CASTFOREM  attributes  the  variability  to  the  observers  (observer-only  draw),  while  the 
JANUS  implicitly  assumes  that  the  target  acquisition  process  is  inherently  stochastic 
(observer-target  draw). 

But,  as  recently  demonstrated  by  David  Dixon  of  TRADOC  Analysis  Command 
(TRAC),  in  many-on-many  engagements,  correlations  between  different  observers  does 
make  a  difference.  Dixon’s  analysis  gave  new  impetus  to  the  resolution  of  the  long 
standing  question  of  the  meaning  of  Poo. 

We  believe  that  we  have  taken  a  large  step  toward  the  resolution  of  this  problem  via 
an  analysis  of  NVESD  observer  test  data.  Our  approach  to  the  problem  evolved  consider¬ 
ably  over  time,  as  we  learned  which  ideas  were  supported  by  the  data  and  which  were  not. 
The  final  formulation  of  our  model  rests  on  the  following  assumptions: 

•  The  probability  that  a  given  observer  will  detect  a  given  target  is  a  function  of 
the  sum  of  two  variables.  One  of  these  variables  is  determined  by  the  partic¬ 
ular  target  signature,  the  other  by  the  individual  observer. 

•  The  test  data  which  we  use  to  determine  these  parameters  employed  a  repre¬ 
sentative  sample  of  the  standard  observer  ensemble. 

•  The  estimate  of  the  target  signature  variable  that  is  computed  in  the  wargames 
is  not  precisely  the  true  value,  but  is  an  estimate  subject  to  a  quantitatively 
known  modeling  uncertainty. 

The  first  two  assumptions  enable  us  to  evaluate  the  relative  importance  of  the 
stochastic  and  the  observer  components  of  the  variability.  Taken  together,  they  support  an 
important  observation:  the  value  of  Poo  is  the  average  of  the  joint  probability  function  over 
the  observer  distribution. 

The  third  assumption  points  to  a  third  source  of  variability  in  the  simulation 
of  target  acquisition — namely,  a  modeling  imcertainty — that  we  need  to  address.  We 
have  recently  published  a  quantitative  analysis  of  the  uncertainty  associated  with  model 


S-2 


predictions.^  This  variability,  it  turns  out,  is  at  least  as  large  as  the  other  two.  It  is 
therefore  important  to  simulate  its  effects. 

Our  final  result  is  a  complete  model  for  the  observer  dependence  of  the  target 
acquisition  probability.  All  required  functions  and  constants  are  provided;  there  are  no  free 
parameters.  The  explicit  code  we’ve  included  is  suitable  for  guiding  the  model’s  imple¬ 
mentation  in  the  wargames.  The  approach  is  similar  to  the  one  now  in  use.  In  the  setup 
phase  of  the  simulation,  random  draws  determine  detection  thresholds.  But  instead  of  a 
single  random  draw,  there  are  three  independent  sets  of  draws: 

•  An  observer-target  draw,  to  account  for  the  stochastic  process; 

•  An  observer  draw,  to  account  for  the  different  level  of  capability  of  the 
observers;  and 

•  A  target  draw,  to  account  for  the  uncertainty  in  the  target  signature  compu¬ 
tation. 

The  essence  of  our  work  is  the  assumption  that  the  target  detection  probability  is 
determined  by  the  sum  of  an  observer  variable  and  a  target  variable.  This  apparently 
innocuous  assumption  generates  most  of  the  simplifications  that  render  the  problem 
tractable.  Therefore,  it  is  this  aspect  of  our  approach  that,  we  hope,  will  receive  additional 
scrutiny.  In  particular,  we  look  forward  to  alternative  formulations  which  do  not  invoke 
this  assumption. 

Meanwhile,  we  offer  our  model  as  a  practical  approach  that  is  sufficiently  similar  to 
existing  approaches  that  it  can  be  implemented  relatively  painlessly,  while  being  as  accurate 
as  it  is  possible  for  such  a  simple  model  to  be. 


^  James  D.  Silk,  Statistical  and  Modeling  Uncertainties  in  the  Thermal  TAMIP  Predictions,  Institute  far 
Defense  Analyses,  IDA  Paper  P-3078, 1995. 
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I.  MOTIVATION 


Stochastic  wargames  rely  on  randomness  to  rescue  them  from  the  plethora  of 
unknowable  variables  that  in  reality  affect  the  outcomes  of  hostile  engagements.  The  intent 
is  that  by  properly  averaging  over  all  possibilities,  the  most  likely  outcome  can  be  deter¬ 
mined.  Unfortunately,  the  transition  from  a  one-on-one  model  (even  if  it  is  well  validated 
and  understood)  to  one  that  accurately  predicts  many-on-many  interactions  is  ambiguous. 

Consider  the  following  rather  contrived  example.  Suppose  that  a  red  commando 
must  avoid  detection  by  both  of  two  blue  sentries  to  accomplish  his  mission.  The  one-on- 
one  probability  of  detecting  the  commando  (averaged  over  all  observers)  is  assumed  to  be 
50  percent.  We  further  assume  that  the  sentries  are  identically  capable,  co-located,  and 
both  fully  vigilant.  What  is  the  commando’s  probability  of  success?  Two  methods  of 
computation  come  to  mind. 

The  assumption  that  is  often  invoked  in  elementary  problems  of  this  type  is  that  the 
“and”  condition — ^that  is,  the  commando  must  evade  both  sentry  one  and  sentry  two — 
means  we  must  multiply  probabilities.  The  use  of  this  recipe  is  based  on  the  implicit 
assumption  that  the  detections  are  statistically  independent  events,  and  are  therefore 
uncorrelated.  That  is,  the  commando  eludes  the  first  sentry  half  the  time,  then  he  eludes  the 
second  sentry  half  of  the  remaining  time,  netting  a  25-percent  probability  of  success. 

But  the  independence  hypothesis  may  not  be  valid  in  fact.  It  is  surely  true  that 
observers  generally  agree  on  the  subjective  evaluation  of  “easy”  and  “hard”  targets. 
Indeed,  it  is  possible  to  argue  (in  the  absence  of  data  to  the  contrary)  that  equally  capable 
observers  are  always  perfectly  correlated  in  their  responses.  After  all,  the  two  sentries  in 
our  example  receive  identical  stimuh;  if  the  sentries  are  deterministic  in  their  behavior,  then 
perfect  correlation  of  their  responses  is  the  logical  conclusion.  In  this  case,  our  commando 
succeeds  50  percent  of  the  time,  because  if  he  eludes  the  first  sentry  then  he  is  certain  to 
elude  the  second. 

(A  third  alternative,  namely  that  the  sentries  are  perfectly  anticorrelated,  is  possible 
in  principle.  In  this  case,  the  commando  never  succeeds  because  if  he  gets  by  one,  the 
other  always  detects  him.  It  is  inconceivable,  however,  that  this  is  a  property  of  real 
human  sentries  who  receive  identical  stimuli.) 
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The  two  assumptions,  uncorrelated  observers  and  perfectly  correlated  observers, 
represent  the  extreme  cases  of  a  range  of  plausible  models  of  human  target  acquisition  that 
are  partly  deterministic  and  partly  stochastic.  It  is  interesting  that  the  two  most  important 
Army  force-on-force  battlefield  simulations,  CASTFORBM  and  JANUS,  use  a  different 
extreme  case.  In  CASTFOREM,  all  of  the  variability  is  assumed  to  be  due  to  variations 
among  the  observers,  who  are  modeled  as  obeying  a  strict  hierarchy;  in  JANUS,  all 
observers  are  statistically  equivalent,  but  independent.  It  is  well  known  that  neither  is 
precisely  correct,  but  up  to  now  no  method  has  been  put  forth  for  determining  whether  one 
is  more  correct  than  the  other. 

It  might  be  hoped  that  the  foregoing  observations  merely  constitute  a  technical 
nuance  that  has  no  practical  import.  Recently,  though,  D.  Dixon  of  TRACAVSMR 
(TRADCXZ!  Analysis  Command-White  Sands  Missile  Range)  clearly  demonstrated  that 
these  two  different  assumptions  can  make  a  vast  difference  in  the  outcome  of  a  simulated 
engagement.  His  examination  of  idealized  forced  march  scenarios  illustrated  that  the 
uncorrelated  model  (the  so-called  observer-target  draw,  which  is  used  in  JANUS) 
generates  a  kind  of  “stochastic  fire  control”  relative  to  the  perfectly  correlated  simulation 
(that  is,  a  simulation  using  an  observer-only  draw,  such  as  CASTFOREM).  Thus,  in  the 
former  case,  one  side  would  be  completely  attritted;  in  the  latter,  the  fronts  would  march 
through  each  other. 

As  part  of  our  Target  Acquisition  Model  Improvement  Program  (TAMIP)  and 
Acquisition  Simulation  Working  Group  (ACQSIM)  activities,  we  have  undertaken  to 
resolve  this  issue.  In  Chapter  n  we  examine  the  test  data  somewhat  qualitatively  to  get  an 
understanding  of  the  nature  of  the  correlations  therein.  In  Chapter  HI  we  propose  a  model 
which  incorporates  an  observer  variable.  Chapter  IV  presents  the  results  of  the  data 
analysis.  We  pursue  two  versions  of  the  analysis;  the  simpler  approach  is  shown  to  give  a 
preferable  description  of  the  data.  Finally,  in  Chapter  V  we  explicitly  define  a  working 
algorithm  for  implementing  a  simulation  of  the  proposed  model. 
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II.  OBSERVER  TEST  DATA 


The  data  set  we  use  herein  is  from  the  NVESD  Phase  lal  and  Ibl  tests."^  We  have 
analyzed  results  from  17  observers  on  313  targets  in  the  former  test,  and  22  observers  on 
275  targets  in  the  latter.  For  the  sake  of  the  discussions  of  this  chapter  only,  we  will 
examine  a  representative  subset  of  the  data  comprising  8  observers  and  10  targets  from  the 
lal  test. 

For  our  purposes,  the  data  consists  of  a  matrix  of  zeros  and  ones.  Each  column  of 
the  matrix  corresponds  to  a  particular  human  observer  who  participated  in  the  test  as  a 
subject.  Each  row  corresponds  to  a  particular  target  in  one  of  a  set  of  images  that  were 
presented  to  the  subjects.  The  matrix  entry  in  ceU  i,j  is  one  if  target  i  was  detected  by 
observer  j,  and  zero  otherwise.  (The  detection  criterion  used  is  100  percent  confidence  that 
the  object  is  a  military  target,  with  no  requirement  for  any  higher  level  of  discrimination.) 
The  matrix  corresponding  to  our  subset  is: 

Matrix  A 

1  1  1  0  0  0  0  0 
10  1110  10 
1  1  1  0  0  0  0  0 
1110  10  11 
1  1  1  0  0  0  0  0 
11110  10  0 
0  0  0  1  1  0  1  0 
1  1  0  0  0  1  0  0 
1110  1111 
11110  10  0 

If  we  average  all  the  entries  in  the  first  column,  we  get  an  estimate  of  the  probability 
that  Observer  #1  can  detect  a  target  from  this  set  (90  percent).  Similarly,  averaging  over 
the  first  row  gives  an  estimate  of  the  probability  of  detection  of  Target  #1  (37.5  percent).  It 
is  hard  to  see  any  correlation  in  the  above  matrix.  But  if  we  reorder  the  rows  and  columns 
by  descending  total  number  of  detections,  we  get  Matrix  B: 


Barbara  L.  O’Kane,  Clarence  P.  Walters,  and  John  D’Agostino,  “Report  on  Experiments  in  Support  of 
Thermal  TAMIP,”  NVESD,  Fort  Belvoir,  VA,  1993. 
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Matrix  B 


1 

1 

1 

0 

1 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

1 

1 

1 

1 

1 

0 

1 

0 

0 

1 

1 

1 

1 

0 

1 

0 

0 

1 

0 

1 

1 

1 

0 

1 

0 

1 

1 

0 

0 

0 

1 

0 

0 

1 

1 

1 

0 

0 

0 

0 

0 

1 

1 

1 

0 

0 

0 

0 

0 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

0 

1 

0 

This  matrix  looks  as  if  it  has  some  correlation  among  observers,  but  it  is  hard  to 
quantify  the  degree  of  correlation.  The  correlation  is  obviously  not  perfect.  If  it  were,  then 
ordering  the  rows  and  columns  would  have  eliminated  all  of  the  islands  of  zeros  in  ones, 
and  vice-versa,  as  in  the  following  fictitious  data: 

Matrix  C 

11111111 
11111110 
1111110  0 
1111110  0 
1  1  1  1  0  0  0  0 

1  1  1  0  0  0  0  0 

1  1  1  0  0  0  0  0 

1  1  1  0  0  0  0  0 

1  1  0  0  0  0  0  0 

1  0  0  0  0  0  0  0 

For  perfect  correlation,  as  in  Matrix  C,  all  of  the  following  statements  hold: 

•  It  is  possible  to  separate  all  the  zeros  from  aU  the  ones  by  drawing  an 
ascending  staircase  through  the  data. 

•  If  observer  A  is  a  better  observer  than  B,  then  A  can  detect  all  the  targets  that  B 
can,  plus  some  others. 

•  Equivalently,  if  target  X  is  easier  than  target  Y,  then  X  can  be  detected  by  aU 
the  observers  that  can  detect  Y,  plus  some  others. 

One  can  verify  by  inspection  that  these  statements  hold  for  the  artificial  data  in  Matrix  C, 
but  not  for  the  real  data  in  Matrix  B. 

Whichever  phrasing  one  prefers,  it  is  clear  that  the  actual  data  does  not  conform  to 
the  perfectly  correlated  hypothesis.  We  now  propose  a  means  of  quantitatively  describing 
the  intermediate  case. 
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III.  MODEL  DEFINITION 


A.  CONCEPTUAL  DESCRIPTION 

The  Thermal  TAMIP  target  detection  model  predicts  target  detectability  as  a  function 
of  a  composite  statistic  (“hne  pairs  on  target”)  that  is  computed  from  target  properties.  We 
refer  to  this  and  other  models  that  do  not  explicitly  include  observer  variables  as  target-only 
models.  The  model  is  stochastic  in  the  sense  that  it  provides  outputs  which  are 
probabilities,  and  these  probabilities  are  appreciably  different  from  zero  or  one  in  a  finite 
range  of  the  target  statistic.  The  actual  relation  between  the  Pd  and  the  target  statistic  will 
be  discussed  later.  For  now,  we  simply  characterize  the  function  as  in  Figure  ni-1:  Pd 
has  a  value  of  0  for  small  values  of  the  target  statistic,  1  for  large  values,  and  an 
intermediate  “stochastic”  region  in  between,  where  it  takes  on  values  significantly  different 
from  0  or  1. 


Figure  III-1.  Probability  as  a  Function  of  a  Target  Statistic 
in  a  Target-Only  Model 

In  the  foregoing  discussions,  we  have  observed  that  there  are  two  possible  sources 
for  this  stochastic  element.  One  source  is  the  variability  between  human  observers.  Since 
individual  observer  properties  are  not  specified,  the  output  must  refer  to  an  average  over  a 
standard  (but  unspecified)  observer  ensemble.  The  other  source  is  the  target  acquisition 
process  itself.  As  we  have  seen,  it  seems  to  have  some  inherent  element  of  randomness. 

Our  approach  is  to  construct  a  model  that  nominally  incorporates  the  observer 
variable.  We  use  the  term  “nominally”  because  we  do  not  seek  to  describe  how  it  might  be 
evaluated  for  a  specific  observer  (e.g.,  a  function  of  eyesight,  experience,  etc.).  That  is, 
we  do  not  pretend  to  know  how  to  compute  the  model  inputs  a  priori.  Instead,  we  simply 
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use  our  construct  to  evaluate  correlations  between  observers  by  taking  an  a  posteriori 
approach:  given  the  results  of  a  test,  what  are  the  most  likely  values  of  the  model  inputs? 
What  can  be  deduced  from  these  values  about  the  correlation  between  observers? 

We  assume  that  the  proficiency  of  the  subjects  in  a  target  acquisition  task  is 
imiquely  determined  by  a  single  variable  s.  Similarly,  we  assume  the  detectability  of  the 
targets  under  a  particular  set  of  observational  conditions  is  uniquely  determined  by  the 
variable  t.  Then,  a  model  estimate  of  the  probability  that  a  given  target  will  be  detected  by  a 
given  observer  must  be  some  function,  call  it  IJ,  of  s  and  t.  We  assume  that  this  functional 
relationship  takes  a  certain  specific  but  very  plausible  form,  but  for  now,  we  discuss  it  in 
the  same  general  terms  that  we  used  for  Pd  above. 

We  note  that  an  elementary  requirement  for  any  such  model  is  that  we  must  by 
definition  retrieve  the  target-only  model  when  we  average  over  the  standard  observer 
ensemble.  In  particular,  this  means  that  we  shall  need  to  find  an  explicit  relationship 
between  our  target  variable  t  and  the  target  signature  variable  that  appears  in  the  Thermal 
TAMIP  model. 

To  achieve  these  ends,  we  consider  three  models:  the  deterministic,  the  stochastic, 
and  the  hybrid.  In  the  figures  associated  with  each  of  these  models.  Figure  III-l  is 
reproduced,  to  emphasize  the  requirement  that  we  connect  back  to  the  “target-only”  model 
that  predicts  performance. 

1 .  Deterministic  Limit 

The  first  case  that  we  consider  is  illustrated  in  Figure  III-2.  The  two  independent 
variables,  s  and  t,  are  the  vertical  and  horizontal  axes,  respectively.  The  shaded  rectangle 
represents  the  ensemble  of  targets  and  observers  that  populate  a  hypothetical  test.  The 
probability  function,  FI,  is  defined  on  the  plane.  This  dependent  variable’s  axis  is 
perpendicular  to  the  page.  The  diagonal  line  divides  the  plane  into  two  regions:  to  the 
upper  right,  where  targets  are  easy  and  observers  are  good,  77=  1;  to  the  lower  left,  77=  0. 

We  call  this  version  of  the  model  “deterministic”  because  there  is  no  region  of  the 
(s,  t)  plane  where  77  is  different  from  0  or  1.  That  is,  for  a  given  observer  and  a  given 
target  there  is  no  random  element.  In  this  case  the  seemingly  stochastic  region  of  the 
“target-only”  model  arises  entirely  from  the  variation  in  the  observer  ensemble. 
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Figure  111-2.  Deterministic  Model 


2 .  Stochastic  Limit 

Figure  111-3  depicts  a  different  extreme.  In  this  case  all  of  the  observers  are 
essentially  equivalent.  This  is  depicted  in  the  vanishing  width  of  the  shaded  rectangle.  On 
the  other  hand,  the  probability  function  /7  now  has  an  intermediate  region — between  the 
diagonal  lines — where  it  takes  on  values  significantly  different  from  0  or  1.  All  of  the 
stochastic  behavior  of  Pj)  is  now  attributed  to  the  inherent  randomness  of  the  detection 
process  itself.  We  therefore  call  this  the  stochastic  limit. 
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3.  The  Intermediate  Case 

We  have  already  observed  that  neither  of  these  cases  is  correct.  The  deterministic 
version  dictates  a  strict  ordering  of  observer  responses;  we  saw  in  Chapter  II  that  this 
ordering  is  violated.  The  stochastic  version  is  also  wrong,  since  we  know  that  there  are 
significant  variations  within  the  observer  population.  Therefore,  a  hybrid  of  these  alterna¬ 
tives  is  required. 

Figure  ni-4  represents  this  hybrid  model.  As  with  the  deterministic  model  there  is 
variation  within  the  observer  ensemble.  And  there  is  a  stochastic  region  for  77  as  well.  The 
stochastic  region  of  Pd  reflects  contributions  from  both  effects.  It  is  important  to  observe 
that  the  fixed  width  of  the  stochastic  region  of  Pd  places  a  constraint  on  the  widths  of  the 
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observer  ensemble  distribution  and  the  stochastic  region  of  77,  since  these  two  functions 
must  convolve  to  form  Pj)  . 
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Targets 

Figure  III-4.  Hybrid  Model 

B  .  FORMAL  DEVELOPMENT  OF  THE  MODEL 

All  of  the  conceptual  ideas  that  we  need  have  been  discussed  above.  We  now  write 
the  formulas  explicitly  so  that  we  can  pursue  the  data  analysis  required  to  validate  the  model 
and  elucidate  the  model’s  implementation. 

The  form  of  the  function  that  relates  the  target  statistic  to  the  probability  of  detection 
in  the  Thermal  TAMIP  Model  is  a  logistic  transform: 

Piyix)  =  Libx)  , 
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where 


b  =  3,  jc  =  In 


^  M  ^ 


y^5oJ 


,  and  L(u)  = 


1 


1  +  exp(-M) 


so 

P  - 

""  m|o  +  a/3 

Although  the  shape  of  this  curve  closely  resembles  the  older  version  of  the  target  transform 
probability  function  (TTPF),  it  has  been  chosen  because  it  is  better  behaved  analytically.^ 
For  the  sake  of  consistency,  we  take  our  function  /I  to  be  a  logistic  function  of  the  sum  of 
the  independent  variables: 

n(s,t)  =  n(s  +  t)~  L(s  +  t)  . 


In  the  data  analysis  that  follows  we  shall  derive  values  of  s  and  t  which  optimize  the  fit  to 
the  field  test  data.  First,  let  us  consider  the  observer  variable  s. 

It  is  not  our  intention  to  attempt  to  predict  observer  performance  based  upon  a  priori 
data  such  as  eyesight,  training,  experience,  etc.  How,  then,  is  the  variable  s  useful?  Hie 
variable  5  is  a  vehicle  to  connect  to  the  standard  observer  ensemble.  That  is,  there  must 
exist — and  we  must  specify! — ^some  cumulative  distribution  function  *P  such  that  ^(s) 
gives  the  fractile  rating  of  the  observer  who  is  characterized  by  5.  In  short,  the  expression 
above  for  /I  is  of  little  use  until  we  specify  Fortunately,  in  the  design  of  the  observer 
tests  which  were  used  for  the  development  and  validation  of  the  Thermal  TAMIP  Model, 
great  care  was  taken  to  use  a  representative  sample  of  observer  subjects.  Therefore,  we  get 
the  necessary  information  almost  for  free. 

To  get  the  functional  form  of  W  we  need  to  explicitly  carry  out  the  averaging  over 
the  observers.  Let  us  define  the  function  as  the  formal  average  of  FI  over  the  standard 
observer  ensemble.  (This  new  function  is  essentially  Pj),  but  we  have  not  yet  connected  t 
and  X  so  we  give  it  a  unique  name  for  now.)  This  average  is  simply  a  convolution  integral 
of  the  observer-target  function  over  the  observer  distribution: 

=  j  y/(s)n(s  +  t)ds  , 

where  yr  is  the  derivative  of  *F. 


^  In  particular,  its  inverse  can  be  written  explicitly.  It  also  has  the  useful  property  that  it  is  related  to  its 
derivative  by  the  formula  L'  =  L(1  -  L)  . 
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]_ex€>{t)  =  L(at).^  Again,  this  is  consistent  with  the  Thermal  TAMIP  definition. 

Then  the  functional  form  of  y^is  determined  by  our  choice  of  the  logistic  function  for  ^and 

n.  Appendix  A  gives  the  method  of  derivation  and  illustrative  computations.  Remarkably 

enough,  the  result  can  be  written  in  closed  form; 

,  ,  1  sin;ra 

y/is)  = - 

iTca  cos  na  + cosh  s 

.T,.  si  1  -if  Tia  ,  s\ 

*P(^)  =  — + — tan  tan — tanh—  - 

2  Tta  V  2  2) 

Note  that  *P,  like  L,  is  easily  inverted.  Thus,  once  we  have  determined  a,  we  can 
readily  construct  a  composite  function  that  connects  the  observer  fractile  rank  (/?),  the 
ensemble  averaged  target  detectability  {Pd),  and  the  observer  specific  detectability  (P): 

P  =  ni^\R),0-\PD))  . 

We  now  return  to  the  target  variable.  To  finally  implement  the  model  into  the 
wargames,  we  need  to  make  the  connection  to  the  “target-only”  model  (in  our  case,  the 
Thermal  TAMIP  Model)  by  establishing  the  relationship  between  t  and  x.  It  seems  reason¬ 
able  at  first  to  identify  with  Pd  and  use  the  definitions  above  to  establish  the  linear 
relationship  between  t  and  x.  This  yields 

b 

t  =  -x  , 
a 

but  there  is  a  subtlety  that  we  must  address  before  we  close  out  this  discussion.  A  problem 

arises  because  the  value  of  x  computed  in  the  Thermal  TAMIP  Model  is  actually  an  estimate 

of  the  true  value  of  the  target  statistic — call  it  x'.  We  have  shown  elsewhere"^  that  the  data 

support  the  assumption  that  x'is  an  unbiased  estimate  of  x  such  that  x  =  x'  +  t],  where  7]  is 

a  Gaussian  random  variable  with  zero  mean  and  0.36  standard  deviation.  See  Figure  III-5 

for  an  illustration  and  further  details.  If  this  “ideal”  value  could  be  determined,  the  “exact” 

/ 

detection  probability  would  be  P D  =  L(b'x')  with  b'=  4.5.  Since  t  is  a  similarly  “ideal” 
quantity  that  has  been  determined  by  inference  rather  than  constructively,  for  the  purposes 
of  wargaming  it  is  more  appropriate  to  use 


^  Since  IJ  depends  only  on  the  sum  of  s  and  t,  the  zero  of  t  was  not  fixed.  By  omitting  a  constant  offset 
term  in  this  equation,  we  are  removing  this  ambiguity  and  constraining  t  =  0  for  50-percent  targets. 
Equivalently,  the  50th-percentile  observer  has  s  =  0. 

^  See  note  3. 
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in  the  simulation. 


Phase  1  Data 


Predictor 

Figure  III-5.  Illustration  of  the  Thermal  TAMIP  Fit  to  the  NVESD  Phase  1  Test 
Data.  The  shallow  curve,  labeled  “opt,”  is  the  b  =  3  fit,  which  is  the  result 
of  minimizing  the  vertical  errors.  The  curve  labeled  “med”  is  steeper, 
corresponding  to  b'  =  4.5,  and  results  from  minimizing  the 
horizontal  errors.  The  other  dashed  curves  illustrate 
the  80-percent  confidence  interval  about  “med.” 

C.  COMMENTS  ON  THE  DISTRIBUTION  FUNCTIONS 

It  was  necessary  to  be  specific  about  the  forms  of  the  cumulative  probability 
distribution  functions  (PDF’s)  that  are  derived  above,  since  our  objective  is  to  implement  a 
calculable  model.  [These  PDF’s  are  'P,  0,  and  77  and  their  associated  differential 
probability  distribution  functions  (pdfs)  y/,  <p,  and  n.]  But  in  retrospect  we  suspect  that  all 
“reasonable”  pdf’s  (i.e.,  bump  in  the  middle,  symmetric,  roughly  exponential  tails)  do 
about  equally  well.  Our  choices  are  a  matter  of  taste  and  historical  context  rather  than  the 
result  of  an  optimization  analysis. 

Similarly,  the  assertion  that  the  target-only  model  is  the  average  over  the  observer 
ensemble  is  almost  without  content.  This  is  self-evident;  there  is  nothing  new  here. 


12 


The  important  element  of  the  model — the  one  that  has  predictive  power  and  that  we 
seek  to  test — is  the  conjecture  that  the  observer-target  model  reduces  to  a  function  of  a 
single  variable,  namely  the  sum  of  the  observer  and  target  statistics.  This  has  an  immediate 
and  specific  consequence:  taking  the  derivative  of  the  defining  relationship  above  gives  the 
relation  among  the  pdf  s: 

^(0  =  J  +  0  ^ 

It  is  now  a  simple  matter  to  show  that  the  model  demands  a  constraint  on  the  widths: 

_2  _  2  ,  2 

^tgt  ^obs  ‘  ^obs*tgt  ’ 

where  cr,^,  =  ,  etc.  Note  that  the  definition  of  P  sets  the  scale  of  s  and  t;  this  fixes 

(^obs-tgt  at  1.814. 

The  data  analysis  will  determine  a  value  of  t  for  each  target  and  a  value  of  s  for  each 
observer.  Thus,  the  above  constraint  on  the  widths  can  be  tested:  the  standard  deviation  of 
the  observers’  s  values  can  be  evaluated  directly,  yielding  The  value  of  a  (which  is 

the  slope  of  the  plot  of  t  against  the  logodds  of  Pd)  can  be  computed  in  a  straightforward 
manner.  Note  that  1/a  measures  relative  to  Oghs.tg,  ■  The  value  of  1/a  will  be  unity  in 
the  stochastic  limit  and  approach  zero  in  the  deterministic  limit. 


13 


IV.  DATA  ANALYSIS 


The  objectives  of  the  data  analysis  are: 

•  To  determine  model  fits  to  the  data  and  to  show  that  the  most  general  fitting 
procedure  is  extremely  well  approximated  by  a  far  simpler  procedure; 

•  To  show  the  consistency  of  the  model  across  different  observing  conditions; 

•  To  calibrate  the  model  by  evaluating  the  population  statistics;  and 

•  To  validate  the  model  by  testing  the  constraint  demonstrated  at  the  end  of  the 
preceding  chapter  on  the  widths  of  the  pdfs. 

To  meet  these  objectives  we  use  the  NVESD  Phase  1  data  set.  For  the  purposes  of 
this  work,  the  NVESD  Phase  1  observer  tests  each  consisted  of  a  set  of  observers 
attempting  to  detect  a  set  of  targets.  Each  observer  was  shown  identical  renderings  of  the 
same  set  of  targets  (but  in  different  orders).  We  let  the  index  i  label  the  M  targets,  and  j 
label  the  N  observers.  Then  the  results  of  the  test  are  described  by  a  rectangular  matrix  of 
M  rows  by  N  columns,  called  P.  The  elements  of  P  are  all  zeros  and  ones  (for  misses  and 
detects);  a  given  element  Pij  indicates  the  result  of  the 7th  subject’s  attempt  to  detect  the  ith 
target.  We  write  Flij  =  n(sj,ti)  to  denote  that  the  matrix  of  model  predictions  for  detection 
probabilities  (on  the  left-hand  side)  can  be  computed  from  the  model  function  if  we  know 
the  V  -I-  M  values  of  s  and  t. 

A.  FITTING  PROCEDURES  AND  RESULTS 

For  a  given  set  of  values  for  s  and  t,  we  can  quantitatively  compare  the  predictions 
to  the  data  via  the  log-likelihood  ratio: 

LLR{s,t)  =  ln(n,. ,.)  +  (1  -  P,.  pin(l  -  n^j)}  . 

u 

The  best  fit  of  the  model  to  the  data  will  correspond  to  the  set  of  sj  and  ti  for  which  this 
expression  is  maximized.^ 


*  Harry  F.  Martz  and  Ray  A.  Walker,  Bayesian  Reliability  Analysis,  John  Wiley  &  Sons,  1982. 
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We  shall  consider  two  approaches  to  this  maximization  procedure.  First,  we  allow 
all  of  the  sj  and  r/  values  to  be  independent  and  unconstrained.  This  means  that  for  the 
NVESD  Phase  lb  data  there  are  275  +  22  =  297  free  parameters.  Second,  we  take  a 
computationally  less  challenging  approach  by  constraining  the  possible  values  of  sj  and  r/  in 
the  manner  described  below.  These  constraints  reduce  the  number  of  free  parameters  to 
two.  This  second  approach  is  somewhat  more  instructive  and  yields  greater  insight  into  the 
model.  Comparing  the  results  of  the  two  fitting  procedures,  it  turns  out  that  the  simpler 
method  costs  almost  nothing  in  terms  of  the  quality  of  the  fit.  We  believe  that  this  approach 
to  the  data  analysis  will  facilitate  widespread  confirmation  of  our  results  and  will  make  the 
model  easily  supportable  and  applicable  to  different  ensembles  of  observers. 

1 .  Unconstrained  Fit 

The  implementation  of  the  first  approach  is  straightforward  in  principle,  but  rather 
daunting  in  practice  due  to  the  high  dimensionality.  The  usual  method  of  choice 
(Mathcad™  for  the  Macintosh)  was  not  practical  at  aU.  A  means  of  searching  on  the  sj  and 
ti  so  as  to  maximize  the  LLR  was  developed  (using  Matlab^w  on  a  Sparc  10)  and  is  given  in 
Appendix  B.  The  complete  set  was  partitioned  into  four  roughly  equal  pieces.  Each  of 
these  pieces  corresponds  to  a  homogeneous  clutter  background,  designated  numerically 
from  1  to  4.  Background  1  was  the  least  challenging,  3  the  most,  with  2  and  4  the 
intermediate  cases.  We  have  broken  the  data  out  in  this  way  in  order  to  stress  the  model:  it 
is  conceivable  that  the  best  observers  in  one  kind  of  clutter  wUl  not  be  the  best  in  another, 
whereas  the  model  would  have  the  ranking  of  observers  be  independent,  (except  for 
statistical  variations)  of  the  nature  of  the  target  acquisition  task. 

We  have  noted  that  /7is  left  unchanged  if  an  arbitrary  constant  is  added  to  aU  the  ss 
and  subtracted  from  all  the  ts.  In  the  previous  chapter  we  formally  removed  this  ambiguity 
by  insisting  that  the  mean  of  .y  be  zero.  As  a  practical  matter,  it  is  easier  to  fix  one  observer 
to  have  y  =  0  and  leave  the  rest  free;  then  implement  the  constraint  after  the  fact. 

2.  Constrained  Fit 

Our  second  approach  takes  as  its  starting  point  the  inverse  of  the  equation  that 
defines  the  logistic  transform: 


n 


where  the  function  on  the  right  is  called  the  logodds  of  n.  In  attempting  to  estimate  the 
values  of  s  and  t  that  fit  a  given  data  matrix  P,  we  might  insert  P  on  the  right-hand  side  for 
77  and  average  over  targets  to  get  the  values  of  s  and  over  observers  to  get  the  values  of  t. 
The  problem  with  this  approach  is  that  the  matrix  P  consists  entirely  of  Os  and  Is,  so  the 
logodds  is  undefined.  Suppose  instead  that  we  effectively  replace  “the  average  of  logodds” 
by  the  “logodds  of  the  average.”  To  do  so,  we  employ  the  so-called  uninformed  estimate 
of  the  probabilities  to  preclude  probability  estimates  that  are  zero  or  one.  That  is,  we  define 


With  each  of  these  probabilities  we  associate  a  logodds: 


=  In 


Tj 

1-7’/ 


It  is  tempting  to  interpret  these  os  and  ts  as  estimates  of  the  ss  and  rs,  which  are  the 
quantities  that  we  seek.  But  this  cannot  be  strictly  correct,  since  these  new  quantities  are 
strongly  biased.  That  is,  all  of  the  observer  odds  could  be  driven  to  any  arbitrarily  large  (or 
small)  value  by  the  inclusion  of  a  suitable  proportion  of  extremely  easy  (or  difficult) 
targets.^  By  hypothesis,  our  ss  (ts)  are  intrinsic  properties  of  observers  (targets)  and 
should  not  be  affected  by  the  composition  of  a  particular  test. 

But  this  is  exactly  the  price  we  expect  to  pay  for  replacing  the  average  of  a  function 
by  the  function  of  an  average.  We  know  that  there  is  a  relationship  between  the  two,  but 
the  relation  depends  on  the  underlying  populations — ^which  for  targets  is  scenario  depend¬ 
ent,  and  for  observers  (at  least  for  now)  is  not  specified.  As  a  way  to  undo  this  damage, 
we  presume  that  the  ss  and  ts  are  linear  transforms  of  the  experimental  logodds  (i.e.,  the  os 
and  Ts),  and  that  the  specific  relationship  is  dependent  on  the  experimental  composition. 
To  be  precise,  we  assume  that  the  unbiased  variables  can  be  written  in  terms  of  the  biased 
ones. 


with  the  understanding  that  the  validity  of  our  conjecture  is  to  be  established  in  the  data 
analysis  which  follows.  As  we  noted  previously,  the  ss  and  ts  are  only  determined  up  to  a 


^  On  the  other  hand,  recall  that  for  tests  of  interest  to  us,  the  observer  sample  is  representative  of  the 
standard  ensemble.  Therefore,  the  t  values  are,  by  hypothesis,  independent  of  the  particular  test. 
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relative  additive  constant.  In  the  above  expression  we  implicitly  define  that  constant  such 
that  our  average  observer  will  have  s  =  0,  and  50-percent  targets  will  have  t  =  0.  Also  note 
that  the  parameter  P2,  which  is  determined  in  the  fit,  is  exactly  the  value  l/a,  the  impor¬ 
tance  of  which  is  discussed  at  the  end  of  the  preceding  chapter. 

We  have  thus  reduced  the  number  of  free  parameters  needed  to  fit  the  NVESD 
Phase  1  data  set  from  hundreds  to  two.  As  a  means  of  fitting  data,  this  choice  is  far 
simpler  than  the  more  general  method  and  is  to  be  preferred,  provided  that  the  quality  of  the 
fit  is  not  much  worse.  As  we  shall  see,  there  is  almost  no  penalty  for  this  simplification. 

The  Mathcad™  script  used  to  perform  these  computations  is  shown  in  Appendix  C. 

3.  Comparison  of  Results 

The  quality  of  the  fits  is  determined  by  the  log-likelihood  ratio  (LLR).  Table  IV- 1 
is  a  comparison  of  the  two  methods.  A  general  rule  of  thumb  is  that  an  additional  degree  of 
freedom  (DF)  yields  significant  improvement  if  it  improves  the  LLR  by  at  least  one  unit. 
We  see  that  in  the  best  case,  the  294  additional  free  parameters  yields  an  improvement  of 
2.49  units.  Another  way  to  look  at  this  is  that  the  specification  of  these  294  extra 
parameters  has  yielded  an  additional  2.49An(2)  =  3.6  bits  of  useful  information.  Clearly, 
the  extra  effort  is  not  worth  the  trouble. 


Table  IV-1.  Log-Likelihood  Ratio  Comparison  Between 
Two  Model  Implementations  by  Clutter  Class 


Clutter 

Class 

LLR 

Method  1 
(296  DF) 

Method  2 
(2DF) 

Difference 

1 

-  402.44 

-  403.40 

0.96 

2 

-  377.93 

-  380.42 

2.49 

3 

-  454.50 

-  456.88 

2.38 

4 

-  473.97 

-  475.87 

1.90 

The  comparison  of  the  fitted  s  values  further  illustrates  the  near  equivalence  of  the 
two  sets  of  results.  Figure  IV- 1  shows  the  results  for  Clutter  Classes  2  and  3.  According 
to  the  table,  these  cases  should  show  the  most  dissimilarity  between  the  two  methods.  In 
fact,  there  is  very  little  difference  between  the  two  methods,  with  most  of  the  differences 
being  associated  with  the  outliers. 
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(O 


s  Values,  Clutter  2 


s  Values,  Clutter  3 


2DF 


2DF 


Figure  IV-1.  Equivalence  of  Observer  Fits  for 
Vastly  Differing  Degrees  of  Freedom 

B.  MODEL  CONSISTENCY 

Our  model  assumes  that  the  competence  of  observers,  which  is  measured  by  the 
variable  s,  is  independent  of  the  specific  target  acquisition  task  that  is  attempted.  Thus,  if 
our  model  is  correct,  we  expect  that  it  will  produce  roughly  the  same  s  value  for  each 
observer,  even  if  we  change  the  level  of  difficulty  of  the  test. 

In  Figure  IV-2  we  compare  the  values  of  s  that  were  determined  for  the  four  clutter 
classes  in  the  test.  (For  reference,  the  average  P£)S  for  each  clutter  class  are  given  in 
Table  rV-2  below.)  The  figure  shows  two  representative  comparisons  of  the  six  possible. 
It  is  clear  that  the  data  follow  the  suggested  trend. 

However,  while  we  have  not  performed  a  fuU  test  of  significance,  we  believe  that 
there  is  sufficient  departure  from  out  prediction  to  indicate  that  out  hypothesis  is  not 
100-percent  correct.  This  is  not  surprising;  we  fully  expect  that  military  observers  are  not 
as  one-dimensional  as  this  model  pretends.  Nevertheless,  the  degree  of  correlation  in  the 
figure  is  encouraging.  We  will  show  that  our  approximation  is  good  enough  for  our 
purposes,  even  though  it  does  not  hold  precisely. 
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Clutter  1 


s  Values 


Clutter  2 


Figure  IV-2.  Correlation  of  Observer  Performance  in  Various  Clutter  Conditions 

e.  CALIBRATION  OF  MODEL  STATISTICS 

The  important  parameters  that  need  to  be  documented  are  the  widths  Gtgu  Oobs,  and 
^obs’tgt  of  the  functions  y/,  <p,  and  n.  Recall  that  by  definition  we  have  set  CTobs^tgt  to 
1.814,  so  we  need  to  determine  the  other  two  from  the  data. 

The  value  of  Otgt  is  determined  by  the  relation  Otgt  =  P2C^obs‘tgt .  which  holds  as  a 
direct  consequence  of  the  two-parameter  fitting  methodology.  The  value  of  (Jobs  is  simply 
the  standard  deviation  of  the  fitted  s  values.  This  follows  from  our  assumption  that  the 
observers  in  the  test  constitute  a  representative  sample  of  the  standard  ensemble. 

Table  IV-2  shows  that  there  is  good  consistency  among  the  standard  deviations  for 
the  four  clutter  classes.  The  global  fit,  to  all  data  simultaneously,  gives  somewhat  smaller 
width  values,  consistent  with  our  observation  that  there  is  slight  reordering  of  observers  as 
the  difficulty  varies.  When  all  levels  of  difficulty  are  folded  together,  the  observer 
ensemble  looks  slightly  more  homogeneous. 


Table  IV-2.  Net  Detection  Probabilities  and  Standard 
Deviations  by  Clutter  Class 


Clutter  Class 

Net  Pd 

^obs 

^tgt 

1 

.77 

1.49  ±  .11 

2.46  ±  .13 

2 

.61 

1.38  ±  .12 

2.38  +  .14 

3 

.35 

1.37  ±  .12 

2.33  ±  .14 

4 

.44 

1.57  ±  .11 

2.56  ±  .15 

global  fit 

.54 

1.27  ±  .05 

2.32  ±  .07 
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D.  VALIDATION  OF  MODEL  CONSTRAINTS 


The  essential  test  of  the  model  is  the  consistency  between  the  widths  of  the  s  and  t 
distributions  that  was  shown  previously:  =  ffobs  +  ^Ibs’tgt  •  This  goes  to  the  central 

assumption  of  the  model — that  target  detection  depends  ons  +  t  only. 

Figure  rV-3  shows  the  result  of  this  test.  The  points  represent  the  data  of 
Table  IV-2.  They  are  individually  consistent  with  each  other  (marginally),  and  with  the 
curve  that  is  generated  by  the  constraining  relation  above.  Collectively,  the  data  points  all 
lie  above  the  curve,  which  is  probably  an  indication  that  our  central  assumption  is  violated 
at  some  level  of  precision.  As  noted  above,  this  should  not  surprise  us.  Rather,  it  is 
remarkable  that  the  model  works  as  well  as  it  does,  given  the  simplicity  of  the  assumptions. 


Widths  of  distributions 


Widths  of  distributions 


2.8 

-  2.6 
C» 

2.4 

2.2 
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i  '  '  ' 

1  '  '  '  1 
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1  '  '  ' : 

T 

! 
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- 
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. . ^ 

. 

- 

♦  2  para  in.  by  ckitler 
□  2  param.  all  duller 
- mode!  constraint 


1.2  1.4  1.6  1.8 


Figure  IV-3.  Consistency  of  Results  for  Different  Clutter  Conditions 
and  Adherence  to  Model  Constraint 

The  global  fit  indicates  that  the  ratio  of  the  widths  ^obs'-^obsugv  ^tgt  dose  to  that 
of  a  3:4:5  right  triangle.  This  is  the  ratio  we  shall  take  as  our  baseline. 

The  knowledge  of  these  parameters  enables  us  to  address  questions  about  individ¬ 
ual  observer  performance.  Figure  rV-4,  for  example,  shows  the  expected  performance  of 
individual  observers,  given  targets  whose  ensemble  averaged  Pd  is  known.  For  example, 
for  a  Pd  =  50-percent  target,  the  90th-percentile  observer  detects  with  75-percent  proba¬ 
bility  while  the  lOth-percentile  observer  has  only  a  25-percent  chance. 
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P  by  Observer 


Performance  by  Observer 


0.1  0.3  0.5  0.7  0.9 

Ensemble  Average  P^^ 


Figure  IV-4.  Performance  of  Individual  Observers  by  Rank 
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V.  SIMULATION 


The  foregoing  methods  and  results  are  summarized  in  this  chapter.  We  do  so  by 
putting  them  together  in  a  working  simulation  that  properly  accounts  for  the  observer  varia¬ 
bility  and  the  inherently  stochastic  component  of  target  acquisition,  as  well  as  the  modeling 
uncertainty.  The  latter  is  a  somewhat  separate  issue,  treated  here  mainly  for  the  sake  of 
completeness  and  consistency. 

We  assume  that  the  wargame  contains  a  target-only  acquisition  module.  By  this  we 
mean  a  model  like  Thermal  TAMIP  that,  for  given  target  signature,  outputs  the  ensemble 
average  probability  that  the  target  will  be  detected.  Having  validated  herein  that  the  target- 
only  probability  is  a  convolution  of  the  observer-target  kernel  with  the  observer  ensemble 
distribution,  there  are  only  three  independent  numbers  that  are  needed  for  the  simulation: 

•  The  width  of  the  target-only  probability  function  in  units  of  the  “observer- 
target”  function.  This  is  the  value  1/a.  Our  baseline  ratio  from  the  data 
analysis  is  c„ts‘^obs-tgr<^tgt  =  3:4:5  ,  so  l/a=5/4. 

•  The  scale  factor  that  connects  our  t  variable  to  the  Thermal  TAMIP  x  variable. 
We  initially  set  the  width  of  the  observer-target  function  to  1.814,  an  arbitrary 
choice.  The  relative  scale  is  defined  by  the  relationship  at-Ex,  where  E  is 
the  Thermal  TAMIP  exponent;  we  have  shown^®  that  the  scale  factor  has  the 
value  £/ a  =  5.625. 

•  The  standard  deviation  of  the  random  variable  associated  with  the  uncertainty 
in  the  target-only  model.  In  the  x  scale,  this  value  is  0.38. 

In  implementing  the  simulation  we  take  the  same  strategy  that  is  used  in  JANUS 
and  CASTFOREM.  The  random  draws  are  done  initially  and  are  used  to  construct  detec¬ 
tion  thresholds.  Then  in  the  course  of  the  wargame,  detection  is  declared  (or  is  not)  based 
upon  whether  the  model  produced  probability  exceeds  (or  does  not)  the  predetermined 
threshold.  The  proper  implementation  of  our  modeling  efforts  is  contingent  upon  correct 
construction  of  the  thresholds. 


See  note  3. 
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We  find  that  we  need  three  independent  draws: 

1 .  An  observer  draw,  to  reflect  the  observer  rank; 

2.  An  observer-target  draw,  to  represent  the  stochastic  component  of  the 
acquisition  process;  and 

3 .  A  target  draw,  to  simulate  the  uncertainty  in  the  target-only  model  prediction. 

The  distribution  functions  are  completely  determined  by  the  three  values  that  we  have  in 
hand.  They  are  all  invertible  in  terms  of  elementary  functions  (we  assume  a  logistic 
distribution  for  the  modeling  uncertainty),  so  the  thresholds  are  easily  obtained.  Once  the 
thresholds  are  computed,  the  game  is  played  precisely  as  it  was  previously. 

The  Mathcad™  script  that  follows  illustrates  the  simulation. 
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File  sim.mcd 

Simulates  the  observer  target  model. 

Equations  with  asterisks  *  are  not  used,  but  are  for  reference  or  clarity. 


Preliminary  function  definitions: 

nrnd(m,s)  :=sv-21n(nid(  1 )  )cos(  nid(  2jt ) )  +  mo 


logit(  X )  :=  (  1  +  exp(  ”  X ) ) 


0 


logoddsf  P )  •=  In 


Gaussian  random  numbers. 
Logistic  transform. 

Inverse  of  logistic  transform. 


We  take  the  relative  widths  of  the  distribution  functions  from  the  data  analysis  of  Chapter  4: 


Assume  baseline 
3:4:5  ratio.  See  text. 

Notational  L 

convenience. 


Width  of  the 
Logistic  pdf. 


By  definition;  this  defines  our  scale. 
®  ObsTgt  L  Below  we  connect  to  the  usual  NVL 
^  Model  variables. 

*^Tgt  ®  ObsTgt 


°  Obs 


Now  need  to  relate  the  s.  t  variables  to  the  scale  of  x  M  '=  1  o  * 

and  consider  uncertainties  in  the  target-only  model:  50  “ 

Option  1 :  Assume  modeling  is  precise: 

E:=3  x(M)  :=ln(-^j  a  * 

50/ 

Option  2:  Assume  modeling  uncertainty  is  as  given  in  IDA  Paper  P-3078: 

E'  ;=4.5  Ox:=0.38  x'(M)  :=x(M)-F  nmd(0.Ox,)a  * 

The  scale  factor  relates  the  two  sets  of  variables;  the  defining  relation  is  a*t=E*x  or  =E'*x'. 
We  choose  Option  2. 

E' 

P  q(  M )  .=  logit(  E'- x'(  M ) )  0  *  scale  •=—  o  Unc  •“  °  x 

3 
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Define  probability  functions: 


Inv_Stochastic_CDF(  P )  logoddsf  P ) 


Unc 

Inv_Unc_CDF(  P  )  ■'= - logoudsf  P ) 


w 


L 


Inv_Observer_CDF(  P )  :  =  -  •  atanh 

a 


Will  use  this  instead  of  inverse  error  function. 
'.an(  7fr(  P  —  0.5  ) ) 


tan(  TtT-O.S ) 


.  r,  .  f  .  /  sin(  Tt  r)  ,  \  , 

test(P)  ■=“ — Inl - —  cosfjt  ru 

a  / 


Alternate  form  for  inverse 
observer  CDF. 


Now  we  can  look  at  the  distribution  functions  and  their  width  parameters. 


P  :=  0.03, 0.04..  0.97 


®  ObsTgt  “  1-814 
a  Tgi  =  2.267 
CfObs  =  1-36 
^  Unc  ~  2.138 


Inv_Stochastic_CDF(  P 
Inv_Unc_CDF{  P ) 
Inv_Observcr_CDF(  P ) 
cest(  P ) 


Following  the  current  practice  in  CAST-CREM  and  JANUS,  we  run  the  simulation  in  two 
steps.  First,  before  the  engagement  begins  we  roll  dice  to  determine  detection  thresholds. 
Then  we  begin  the  game,  declaring  detections  when  the  detection  probability  from  the 
target-only  model  exceeds  the  preset  threshold. 


26 


First  do  the  preliminary  draws.  Need  obs,  tgt,  and  obsMgt  draws... 


n  obs  27 


n_tgt  25 


i_obs  :=  0..  n_obs  —  1 


i_tgt  -=0..  n_tgl—  1 


Observer_Ftactile-  ^  ^ 


This  determined  the  observer's  tractile  rating. 


Target_FractiIej  mdt  1 ) 


Stoct5astic_Fractile-  j  mcl(  1 ) 


This  is  the  tractile  rank  ot  the  target-only  model 
uncertainty. 

And  this  is  the  inherently  stochastic  piece  ot  the 
target  acquisition  process. 


Do  the  inverse  transforms.  These  manipulations  just  extend  the 

rows/cols  so  all  matrices  ate  the  same 

_ j  dimension: 

sl_thresh  :=  Inv_Observer_CDF(  Observer_Fractile )  cij  jgi  •“  1  s_ihresh  sl_thresh  cl 


tl_unc  •=  Inv_Unc_CDF(  Target_Fractile ) 


C2j  1  t_unc  •=c2  ll_unc 


sl_thresh  Inv_Stochasuc_CDF(  Stochastic_Fractile ) 


Verity  the  statistics  generated  by  the  random  draws.  Note  the  sample  is  small. 


®  Obs  ~ 

^  Unc  “  2.138 


stdev(  sl_thresh )  =  1.351 
stdev(  tl_unc )  =  2.022 


o  ObsT'U  =  ^  ^ stdev(  st_lhresh'^‘-‘®‘^  )  =  1.767 

°  n  tgt  • 


«_tgt 


Construct  tinal  threshold  values. 

i_Uiresh  .st_thresh  —  s_thrcsh—  t_unc 

- » 

P_thresh  logiit  a- 1_ thresh )  a  *  Just  a  reminder. 

x_thresh  '=  — —■  t_thresh  Transform  to  x-space  tor  direct  comparison  to 

Thermal  TAM  IP  output. 
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Now  play  the  game.  We  simulate  acquisition  for  all  observer-target  pairs: 


- p 

D  •=  (  x>  x_thresh ) 


The  results  of  the  engagement: 

1  lOlOOOOOlOOO 
1  1  1  lOOOOOlOOO 

lllllOlOOOOOO 
1  1  1  I  1  1  1  0  1  1  1  1  0 

0111111011100 
1011111011100 
0101111011100 
0010111011100 
0010001011100 
0110011011111 
0010001011111 
0000000011100 
0000000011  101 
0000000011111 
0000000011111 
0000001  Olllll 
0000000001  100 
0000000001  1  10 
0000000001  100 
0000000001  100 
0000000001000 
0000000000000 
0000000001  100 
0000000000000 
0000000000000 
0000000000000 
0000000001000 


Two  lines  5  km  long,  0.5  km  apart. 


Signatures  otherwise  identical. 
Compare  to  threshold  values. 
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GLOSSARY 


ACQSIM 

Acquisition  Simulation  Working  Group 

ARPA 

Advanced  Research  Projects  Agency 

DF 

degree  of  freedom 

LLR 

log-likelihood  ratio 

NVESD 

Night  Vision  and  Electronics  Systems  Directorate 

pdf 

probability  distribution  function 

TAMIP 

Target  Acquisition  Model  Improvement  Program 

TRAC-WSMR 

TRADOC  Analysis  Command  -  White  Sands  Missile  Range 

TTPF 

target  transfer  probability  function 

GL-1 


APPENDIX  A 


DERIVATION  OF  OBSERVER  FUNCTION 


A-1 


APPENDIX  A 

DERIVATION  OF  OBSERVER  FUNCTION 


The  form  of  the  probability  distribution  function  (pdf)  that  describes  the  observer 
ensemble  is  uniquely  determined  by  our  choice  of  the  logistic  function  for  the  target-only 
and  observer-target  probability  functions.  In  this  appendix  we  explicitly  derive  this  form  of 
the  observer  ensemble  distribution  function. 

We  emphasize  that  the  precise  forms  of  the  functions  that  we  introduce  are  not  of 
fundamental  importance.  Since  the  current  state  of  the  art  does  not  support  excursions  into 
the  tails  of  the  probability  distributions,  one  "S-shaped  curve"  is  more  or  less  as  good  as 
another,  as  long  as  the  important  parameters  (namely  the  position  and  width  of  the 
transition  region)  are  accurately  represented.  We  have  been  explicit  because  it  is  necessary 
to  generate  a  concrete  model  which  is  suitable  for  implementation  into  the  wargames  and  at 
the  same  time  is  consistent  with  the  existing  suite  of  model  results.  Additional  work  on  the 
specific  functional  forms  may  lead  to  marginally  better  fits,  but  will  likely  be  both 
intellectually  and  practically  unproductive. 

The  logistic  function  that  defines  the  observer-target  probability  function  is 

1  f  3.x  ^ 

Fa(x)  =  A(ax)  =  -  l-i-tanh—  , 

2  V  2  y 

and  a  similar  definition  holds  for  the  target-only  function,  Ff,(x) .  The  defining  relation  is  a 
convolution  integral;  our  problem  is  to  find  ^(x)  such  that 

Fb(x)=  f“’K,,b(x-x')F,(x')dx'  . 

J—oo 

It  will  be  convenient  to  use  Fourier  transforms,  so  we  need  to  work  with  normed 
functions.  Taking  the  derivative  yields 

fb(x)  =  |  Ka,b(x-x')fa(x')dx'  , 

where 

fa(x)  =  ^Fa(x)  = -^sech^  ^ 
dx  4  2 


A-3 


and  f  is  normed  to  unity. 


Taking  the  Fourier  transform  and  solving  for  the  transformed  kernel  gives 

K.,b(^)  =  fb(?)/f.(«), 

where  the  hat  denotes  the  Fourier  transform,  ant  it  can  be  shown  that^ 

fa(0  =  2— csch-^  . 
a  a 


Then 


K..b(«) 


■  b 

a  Sinn — 


bsinh  — 
b 


and  the  inverse  transform  yields^ 


Ka,b(x) 


b  sin;rr 


2  ;r  r  cos  Trr  +  cosh  bx 


where  r  =  b/a . 

The  cumulative  distribution  function  (cdf)  can  also  be  written  in  terms  of  elementary 
functions^: 

r  ^  11/^  jTT  bx 

Ha,b(x)=  Ka,b(x')dx'  =  -  +  — atan  tan— tanh  — 

•/— oo  X  Trr  \  ^  2^ 

Note  that  the  cdf  depends  only  on  the  ratio  r  =  b/a,  as  we  would  expect.  The  final  result 
for  the  observer  ensemble  cdf  is  then 


where  b  = 


1.814 

0’s 


•Fr5;=Hi,br5;  , 


and  CTg  is  the  standard  deviation  of  the  observer  s  values,  as  determined  in  the  experimental 
data  analysis. 


1  A.  Erdelyi  et  al.,  "Tables  of  Integral  Transforms,"  Volume  1,  McGraw-Hill,  1954,  p.  30,  equation  (2). 

^  Ibid.,  equation  (6). 

3  I.S.  Gradshteyn  and  I.  M.  Ryzhic,  "Tables  of  Integrals,  Series,  and  Products,"  Academic  Press,  1965, 
p.  108,  equation  2.444.1. 
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The  following  two  Mathcad'^'^  scripts  verify  these  results  numerically.  The  file 
conv.mcd  explicitly  carries  out  the  convolution  and  compared  to  the  expected  result,  then 
explores  the  shape  of  the  function  and  verifies  the  quadratic  relation  among  the  width 
parameters.  The  file  cdf.mcd  confirms  that  H  is  indeed  the  cdf  associated  with  K. 
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conv.mcd 


Verify  that  a  kernel  of  the  form  K  convolved  with  f  results  in  a  rescaling  of  f. 

from  Bateman's  Tables  of  Integral  Transforms,  A.  Erdelyi,  page  30  equations  (2)  and  (6). 


f(  a,  X ) 


Define  f.  Note  a  is  inverse  length  scale. 


K(  a,  b,  X ) 


Define  K. 

Must  have  b<a  because  output 
f  is  wider  after  convolution. 


a  1  b  •'=  .55  Select  the  scale  factors. 


C(y) 


•20 

K(  a,  b,  y  —  X  )•  t(  a,  x )  dx 

-20 


Define  the 

convolution 

integral. 


Check  normalization: 


Check  the  functions: 

t  :=o.o.i..- 

b 


Do  the  computation  for  a  set  of  N  points. 


10 
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Rewrite  functions  to  parameterize  by  standard  deviation.  Check  relation  among  widths. 


Logistic  PDF ; 


s  := 


r20 

x^- 1(  1 ,  X )  dx 

J  J-20 


s  =  1.814 


L(  a, T,  X )  :=  K 


i  :=0..  7 


a  :=0.05-2‘ 


-  V 


V.  •  = 


l  +  (Oi)^ 


r  lO  i; 


-lO-T. 


X  •  L(  1 ,  t  - ,  X  )  dx 


To  good  accuracy,  the  widths 
add  in  quadrature.  Works  best 
for  =  input  widths. 


X  :=  0.0.1..  4.9 


These  two  functions  have 
the  same  width. 

The  kernel  has  a  sharper 
peak  and  a  longer  tail  than 
the  logistic  pdf. 
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Look  at  the  variation  in  shape  of  the  kernel  function  as  the  parameters  are  varied  subject  to  the 
constraiunt  of  constant  width... 


KW(a.b.x)  ;=K 


2  _  k2  2 

s  —  b  -a 


\ 

,  b,  X 

/ 


a  ;=  1 

m  :=0..  N—  1 
,  .  m  +  0.5  s 


N  :=5 

Argument  of  K  must  be  real,  so  there  is  an  upper  limit  on  b. 


X  :=0,0.1..  7.9 
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file  cdf.mcd  Jim  Silk 

Closed  form  of  the  cumulative  distribution  function  for  the  kerne!  that  connects 
scaled  logistic  functions. 

The  kernel  K  is  defined  in  the  file  conv.mcd. 

Our  CDF  is  defined  as  follows: 


CDF(  a,b,x) 


•  X 

K(  a,  b»  x')  dx'a 

J*  oo 


Equivalent  definitions  of  the  kernel  function: 


Result  is  from  Bateman's  Tables  of  Integral  Transforms. 
A.  Erdelyi,  Volume  2  page  30  equations  (2)  and  (6). 


u/  ^  1  Sm(7CT) 

b(r.  z) - • - 

2-7C'r  cosh(  z)  +  cos(  ;c-r) 


K(  a.  b.  X ) 


Ko,j{  a,b,x) 


X  :=  0,0.1. .4.9 


K(  1.2 
•^old< 


X 


Check  equivalence. 


h(  r,  x')  dx’ 


The  integral  form  (definition)  of  the  CDF. 


H  j(r,  X)  :=  I 

H(  r  X )  *  atanf  sin(7c  r)  \  This  form  has  branch  cut  problems  when 

jr  r  \cos(  Jt  r)  +  exp(  “x )/  the  sine  is  negative,  so  use  angle  function. 

H(r,  X)  :=— anglef  cos(  Tcr)  +  exp(-x),  sin(  JtT) )  Easy  to  check  that:  h(r.  x)  :=— H(r,  x)a 
jt-  r 

CDF(a,b,x)  :=H^-^,b  xj 


dx 

This  form  is  just  as  good; 
see  Gradshteyn  2.444.1. 


X  :=-3..3 


1 


H  {(  .  7 ,  X  ) 


^  ncw^  •  ^  )  0 .5 


H(  .7.x) 


0 


”4-20  2 


Closed  forms  both  agree  with 
numerical  integration. 


X 


4 
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Matlab™  CODE  USED  TO  PERFORM 
UNCONSTRAINED  FITS 
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APPENDIX  B 

Matlab™  CODE  USED  TO  PERFORM 
UNCONSTRAINED  FITS 


load  blmat 
pl=blmat (1:69,  : )  ; 
p2=blmat (69+1 :2*69,  : )  ; 
p3=blmat (2*69+1:3*69,  :)  ; 
p4=blmat (3*69+1:275,  :)  ; 

p=pl;  %Select  clutter 

po=mean (p) ' ; 
pt=mean (p ' ) ' ; 

[nt, no] =size (p) ; 

lo=log ( (0 . 001*ones (size (po) ) +po) . / (1 . 002*ones (size (po) ) -po) ) ; 
lt=log ( (0 . 001*ones (size (pt) ) +pt) . / (1 . 002*ones (size (pt) ) -pt) ) ; 
l=[lo;lt] ; 

1=  [1 . 925*lo-2 . 613* (ones (size (lo) )); 1 . 366*lt ]  ;  %From  2-par  fit 
n_rep=199; 


clear  tmp; 
clear  tmpO; 
tmp=l; 
tmp0=tmp; 
llrt=llr(l,p) ; 

%See 

function 

def . 

below. 

for  i=l:n  rep; 

ld=demean (tmp ( : , i : i) , no,  nt)  ; 

%See 

function 

def . 

below. 

l=newton (Id, po, pt ) ; 

%See 

function 

def. 

below. 

tmp ( : , i+1 : i+1 ) =1; 

tmpO ( : , i+1 : i+1) =tmp ( : , i+1 : i+1) -tmp ( : , i : i) ; 

llrt  (i+1) =llr (l,p) ; 

end; 

sl=tmp (1 :no, : ) 
tl=tmp (no+1 :no+nt, : ) ' ; 
llrl=llrt ' ; 

save  sl.dat  si  -ascii  -tabs 
save  tl.dat  tl  -ascii  -tabs 
save  llrl.dat  llrl  -ascii  -tabs 
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function  lnew=newton (1, po, pt ) ; 

% 

no=size (po, 1) ; 

nt=size (pt, 1) ; 

lo=l ( 1 : no) ; 

lt=l (no+1 :no+nt) ; 

tmp=exp  (-It) *exp  (-lo ' ) ; 

q=l . /  (ones (size (tmp) ) +tmp) ; 

qo=mean (q) ' ; 

qt=mean (q' )  '  ; 

fdl= [po-qo;pt-qt]  ; 

sdO=q. * (ones (size (q) ) -q)  / 

sqo=sum (sdO)  '  ; 

sqt=sum (sdO  ' )  ' ; 

sdl=- [diag (sqo)  sdO';sdO  diag(sqt)]; 
sd=sdl (2 :nt+no, 2 :nt+no) ; 
fd=fdl (2 :nt+no)  ; 

11=1 (2 :nt+no) ; 
lnewl=ll-inv (sd) *fd; 
lnew= [1(1:1) ; Inewl  ]  ; 


function  ld=demean (1, no,  nt )  ; 

% 

lo=l (1 :no) ; 
lt=l (no+1 :no+nt) ; 
m=mean (lo) ; 

ld= [lo-m*ones (size (lo) ) ; lt+m*ones (size (It) ) ] ; 


APPENDIX  C 


Mathcad™  SCRIPT  USED  TO  COMPUTE 
TWO  PARAMETER  FITS 
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New  Obs  •  Tgt  file. 

Calculates  two  predictors  from  observer  and  target  probability  estimates. 
Then  do  logistic  fit  to  optimize  the  combination. 

PP  ■=  READPRN(  matrix )  rows(  PP )  =  275 

N  rows  •=  PP )  N  ^.0,3  :=  cols(  PP)  ...  ^  ^ 


j  :=o..N 


Do  Clutter  Class#!:  P.  .  :=PP.  j.n«o  • 

K  J  ITU-  oy,  j 


cols 


Nfows  N^.0,3  :=cols(P) 

i  •—  0..  N  “1  j  •-  0..  N  „i.  1 


NobS;  :=5]i«Pi.j<0-0.1) 
j 

i 


^  rows  " 


N  u  =22 
^  obs,  ^ 


Ncois=22 


^tgto  =  69 


Nnet=  I-518'IO^ 


tgt;  ■“ 


N 


ObS; 

'  J 


•I]i«Pi.j<0-0-Pi.j)  PobSj  -^E^^Pi.j^O-O’Pi.j 

^  tgl; 


net 


P  net  =  0.769 


limit(x)  •=  if^lxl  <  .99,  X,  .99- 


r) 


lo(  X  )  '^2'  aianh(  limit(  2-  x  —  1 ) ) 


logodds(  X ) 

Iogit(  X )  0.5*  ( tanh(  0.5*  x )  +  1 ) 


DATA.  1  .  , ,  rt’“P- 

I  +  j  -  N  rows’  ®  * 

j 

k  —0..  N  N  —  1 

DATA,  t  .  ,-=10, 

1  T  J  -  N  rows’  1  1 

^  max  ^  0^  0, 0, 1 ) 

DATA.  .  -.  •=  loj 

1  +  J  -  N  rows*  ^  1 

k 

DDATA  :=cson(  DATA.O) 


sbar  mean(  DDATA*^  *  ^  ) 


sbar  =  1.347 


idaij^  J  1=  DDATAj^  I  “  sbar 
idatj^  Q  •“  DDATAj^  2 
•=  DDATA,, _0 


mean' 

(lo(Ptgl)) 

=  1.818 

stdevj 

^lo(  P  tgt  )j 

=  2.239 

meani 

l‘°<Pobs)J 

'  =  1.347 

stdev| 

,‘°(Pobs)j 

=  0.772 
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nclat  ’=  rows(  idat ) 


ndai  =  1.518‘ 10^  ^obs'”* 

npar  1=  cols(  idat )  —  1  npar  =1  P  ddat 

i  •=  0..  ndat  —  1  idat^  q  •-  la  bar  ^  bar  “  0-769 

k:=0..npar  unit^  :=  1 


logitsC  p ) 

pcEd(  Igt ) 


•-  idat-p 

:=  l^l+tanh(-|^jj 


Q(  P)  •■=pred(  logits(  p) ) 


Q2(q)  :=(q-(  1  -  q)) 

T 

xtnd(  q)  .=  q-unit 

- » 

Idt(QQ)  :=(idat  QQ) 

Idai(p)  :=Idt(xtnd(Q2(Q(p)))) 


LLR0(  q)  :=  N  obs'Zl  P- ln(  q)  +  (  1  -  P  )•  ln(  1  -  q) ) 
LLR(p)  :=LLR0(Q(P)) 

FD(P)  :=No,,3-[idaJ-(P-Q(p))] 

SD(  P )  :=  -  N  (idat^-  ldat(  P )) 
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SAT  :=  LLRW  0.0(X)1  +  .9998-  P ) 


SAT  =  -0.152 


:=P  n  :=4  n  :=0..  N 

g<n+l>  ;::3<n>  _  sd(  ) 

p;_B<N+l>  LLR(p)  = -403.4 

D  :=  2-  (  SAT  -  LLR(  P ) )  D  =  806.496 


1 

2 

1.239 

1.83 

1.345 

1.912 

1.357 

1.924 

1.357 

1.924 

1.357 

1.924 

n  0“  1 


cnonn(  “  y  2(  D,  ndat  —  npar  —  1 ) )  =  1 
y  2(  D,  ndai  —  npar  —  1 )  =  “14.892 


FD(p)  = 


t 9.886- 10* 
■1.036M0' 


^  /-287.734  89.387  \ 

SD(p)  = 

\  89.387  -74.993/ 


I  :=-SD(p)  I  :=i 


/0.006 
V  0.007 


0.007 

0.021 


X  *=  eigenvaJs(  I ) 


<  k  > 

s  -^eigenvecC  ^^X^) 


X  = 


Pk 

SE, 

WakJj^ 

1.357 

0.074 

18.271 

1.924 
_ i 

0.146 
_ 1 

1 

13.225 

T 

s  -s 


(: 

“) 

T  /  0.003 

s  Is  = 

“  ] 

\0 

1/ 

\o 

0.024/ 
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